Move to ES bridge API #336

mashhurs · 2025-07-10T22:16:19Z

Description

This PR highlights the initial changes required to move to the Bridge Stable API. However, this doesn't mean these changes will be enough for the plugin to be a in normal working state. Rather, this changes highlight how does current bridge Spable API state align with plugin requirements. Based on the discussions on these changes, we may need to adjust the plan (sub-tasks).

Closes [Move to ES bridge -> ES scope] Evaluate current status of Stable API based on plugin usage #331

Next steps

Get review and land [Logstash] Move elastic_integration plugin usage to ES logstash-bridge. elasticsearch#131486
Adapt Java/Ruby unit test cases to move to bridge changes

PLEASE IGNORE:

gradle.build changes which are already counted in [Move to ES Bridge] Build logstash-bridge and include in the plugin #330, this is a requirement to call Bridge Stable API interfaces.

src/main/java/co/elastic/logstash/filters/elasticintegration/EventProcessor.java

src/main/java/co/elastic/logstash/filters/elasticintegration/EventProcessorBuilder.java

src/main/java/co/elastic/logstash/filters/elasticintegration/IngestDuplexMarshaller.java

src/main/java/co/elastic/logstash/filters/elasticintegration/geoip/GeoIpProcessorFactory.java

src/main/java/co/elastic/logstash/filters/elasticintegration/ingest/PipelineProcessor.java

src/main/java/co/elastic/logstash/filters/elasticintegration/ingest/RedactPlugin.java

mergify · 2025-07-10T23:14:21Z

This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

src/main/java/co/elastic/logstash/filters/elasticintegration/PipelineConfigurationFactory.java

src/main/java/co/elastic/logstash/filters/elasticintegration/IngestDuplexMarshaller.java

gradle.properties

…s. Removes required upwrap interface since upstream moved to default method. Utilizes metadata version field since upstream made it accessible.

…sor) changes.

…nBridge

build.gradle

gradle.properties

mashhurs · 2025-08-08T18:47:29Z

src/main/java/co/elastic/logstash/filters/elasticintegration/geoip/ValidatableIpDatabase.java

-
-import org.elasticsearch.ingest.geoip.IpDatabase;
-
-public interface ValidatableIpDatabase extends IpDatabase {


No direct usage.

mashhurs · 2025-08-08T18:48:06Z

src/main/java/co/elastic/logstash/filters/elasticintegration/ingest/RedactPlugin.java

-
-import java.util.Map;
-
-public class RedactPlugin extends Plugin implements IngestPlugin {


moved to the upstream bridge.

src/main/java/co/elastic/logstash/filters/elasticintegration/util/PluginProjectResolver.java

…s to logstash-bridge.

- `new ${BRIDGE}(...)` -> `${BRIDGE}.create(...)`: use bridge-provided factory methods - `${INTERNAL}Bridge.${NESTED}` -> `${INTERNAL}${NESTED}Bridge`: extract nested bridges to top-level - `${BRIDGE}.AbstractExternal` -> `AbstractExternal${BRIDGE}`: extract nested "AbstractExternal" base implementations to top-level

align bridge to yaauie/elasticsearch@94d58d47cd

…tions.

yaauie

Overall LGTM.

We need to first merge the upstream PR on elasticsearch elastic/elasticsearch#132595 (review) and then ensure we're pointing at the correct ELASTICSEARCH_REPO and ELASTICSEARCH_TREEISH in gradle.properties.

src/main/java/co/elastic/logstash/filters/elasticintegration/geoip/GeoIpProcessorFactory.java

build.gradle

elasticmachine · 2025-09-05T21:42:20Z

💚 Build Succeeded

Buildkite Build
Commit: 65d6009

History

💚 Build #645 succeeded d17ad2e
💚 Build #644 succeeded 7c0b6c7
💚 Build #643 succeeded 2dda5e1
💔 Build #642 failed 2dda5e1
💔 Build #641 failed 16dfbdb
💔 Build #640 failed 53ded41

yaauie

👍🏼 Looks good. It's been a long road but it will be very good to have this stable bridge in place.

mashhurs commented Jul 10, 2025

View reviewed changes

mashhurs changed the base branch from move-to-es-bridge-api to main July 10, 2025 23:13

mashhurs added the backport-skip Skip automated backport with mergify label Jul 10, 2025

mashhurs force-pushed the move-to-es-bridge-investigation branch from 0707b1b to bc9af21 Compare July 10, 2025 23:16

mashhurs commented Jul 10, 2025

View reviewed changes

src/main/java/co/elastic/logstash/filters/elasticintegration/PipelineConfigurationFactory.java Outdated Show resolved Hide resolved

mashhurs changed the title ~~[Move to ES bridge API] Investigation~~ DO NOT MERGE: [Move to ES bridge API] Investigation Jul 10, 2025

mashhurs mentioned this pull request Jul 10, 2025

[Move to ES bridge -> ES scope] Evaluate current status of Stable API based on plugin usage #331

Closed

mashhurs commented Jul 16, 2025

View reviewed changes

src/main/java/co/elastic/logstash/filters/elasticintegration/IngestDuplexMarshaller.java Show resolved Hide resolved

mashhurs force-pushed the move-to-es-bridge-investigation branch from 18333c6 to 5a96c8c Compare July 17, 2025 20:06

mashhurs mentioned this pull request Jul 17, 2025

[Logstash] Move elastic_integration plugin usage to ES logstash-bridge. elastic/elasticsearch#131486

Merged

mashhurs changed the title ~~DO NOT MERGE: [Move to ES bridge API] Investigation~~ [DO NOT MERGE]: Move to ES bridge API Jul 18, 2025

mashhurs force-pushed the move-to-es-bridge-investigation branch from 9820699 to 9bdad09 Compare July 22, 2025 19:44

mashhurs changed the title ~~[DO NOT MERGE]: Move to ES bridge API~~ Move to ES bridge API Jul 23, 2025

mashhurs commented Jul 23, 2025

View reviewed changes

gradle.properties Outdated Show resolved Hide resolved

mashhurs linked an issue Jul 24, 2025 that may be closed by this pull request

[Move to ES Bridge -> plugin scope]: Move classes to Bridge API #333

Closed

mashhurs mentioned this pull request Aug 5, 2025

Sync up with upstream ES main: apply project ID resolver to ScriptService. #357

Merged

mashhurs added 5 commits August 7, 2025 16:58

Replace ES classes with their Bridge Stable API pairs.

6c55a10

Points to personal ES repo which provides Stable API interface update…

52d8dd0

…s. Removes required upwrap interface since upstream moved to default method. Utilizes metadata version field since upstream made it accessible.

Plugins and processors moved to the bridge

294ae12

Apply conceptually modification (internal, external and bridge proces…

71cafcb

…sor) changes.

Utilize moved processor definition constants in the IngestCommonPlugi…

c2446a2

…nBridge

mashhurs force-pushed the move-to-es-bridge-investigation branch from 09c332c to 9b0b557 Compare August 7, 2025 23:58

Use logstash-bridge GeoIP interfaces.

53ded41

mashhurs force-pushed the move-to-es-bridge-investigation branch from fa3468c to 53ded41 Compare August 8, 2025 18:40

mashhurs commented Aug 8, 2025

View reviewed changes

mashhurs linked an issue Aug 8, 2025 that may be closed by this pull request

[Move to ES Bridge -> plugin scope]: Apply GeoIP changes in the plugins #354

Closed

Apply interface changes in unit tests.

16dfbdb

mashhurs marked this pull request as ready for review August 11, 2025 16:51

Move RefCountingRunnable, FailProcessorException and Releasable usage…

2dda5e1

…s to logstash-bridge.

mashhurs linked an issue Aug 11, 2025 that may be closed by this pull request

[Move to ES bridge -> ES scope] Introduce wrappers for missed interfaces and close the gap #335

Closed

mashhurs removed a link to an issue Aug 29, 2025

[Move to ES bridge -> ES scope] Introduce wrappers for missed interfaces and close the gap #335

Closed

align bridge to yaauie/elasticsearch@94d58d47cd

fb94ed0

mashhurs mentioned this pull request Sep 2, 2025

Review move to bridge PRs. #366

Closed

yaauie and others added 3 commits September 3, 2025 15:21

Merge pull request #1 from yaauie/bridge-alignment-review

7c0b6c7

align bridge to yaauie/elasticsearch@94d58d47cd

Point to PR base ES upstream repo and branch, minor source simplifica…

d17ad2e

…tions.

yaauie reviewed Sep 5, 2025

View reviewed changes

src/main/java/co/elastic/logstash/filters/elasticintegration/geoip/GeoIpProcessorFactory.java Outdated Show resolved Hide resolved

build.gradle Show resolved Hide resolved

Point to ES main after merging upstream ES changes.

65d6009

mashhurs requested a review from yaauie September 5, 2025 22:44

yaauie approved these changes Sep 8, 2025

View reviewed changes

mashhurs merged commit fd524b7 into elastic:main Sep 8, 2025
4 checks passed

mashhurs deleted the move-to-es-bridge-investigation branch September 8, 2025 17:03


		import org.elasticsearch.ingest.geoip.IpDatabase;

		public interface ValidatableIpDatabase extends IpDatabase {


		import java.util.Map;

		public class RedactPlugin extends Plugin implements IngestPlugin {

Uh oh!

Move to ES bridge API #336

Move to ES bridge API #336

Uh oh!

Conversation

mashhurs commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Next steps

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mashhurs Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

mashhurs Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yaauie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Sep 5, 2025

💚 Build Succeeded

History

Uh oh!

yaauie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mashhurs commented Jul 10, 2025 •

edited

Loading